Automating execution of arbitrary graphical interface applications

ABSTRACT

Interaction with an application interface is monitored and an image of the interface prior to a user interaction is compared with an image after the user interaction to identify a region of interaction. The comparison may be carried out by detecting that a threshold change has occurred using a computationally inexpensive algorithm and then drawing boundaries for the region using a more computationally intensive algorithm. A marker in a UI evolution timeline is associated with an image of the region and coordinates of the interaction normalized and relative to the region. An interaction interval and a profile of change are also stored in association with the marker. For playback, a script can be constructed from the timeline to control automated execution of the application by matching a current interval and profile to the interval and profile for a recorded interaction, and automatically applying that interaction to the current user interface.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of copending U.S. patent applicationSer. No. 13/180,015 filed Jul. 11, 2011.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to computer software, and moreparticularly to a method of characterizing user interactions with agraphical user interface (GUI) of a program application for later use inautomated execution of the GUI, such as in quality assurance testing.

2. Description of the Related Art

Computers and computer software have become ubiquitous in modernsociety, whether in powerful systems such as servers and technicalworkstations, or simpler hand-held devices such as smartphones. It isoften desirable to automate the execution of a software application fora variety of purposes including testing, development, demonstration, orbusiness process automation. Automated execution of an application istypically carried out using a script prepared by a designer or tester,i.e., a series of commands at the user interface level of theapplication. Early computer systems and applications used rudimentarycommand line (text) interfaces and basic script formats, but modernsystems provide a more intuitive approach with graphical user interfaces(GUIs) that can be manipulated by other means such as a specialkeystroke, a keystroke combination, a graphical pointing device (mouse),or a touchscreen.

The advent of GUI-based applications has greatly increased userfriendliness and acceptance of software applications but has lead to aconcomitant increase in the complexity of the interface. Not only hasthe interface become populated with a larger number of interface objectsand varieties of such objects (e.g., text fields, action buttons, checkboxes, radio buttons, hyperlinks, etc.) but there is also a heavierreliance on these features of the GUI. This interface revolutionpresents increasingly difficult challenges for automated execution ofsoftware applications, particularly in the context of quality assurance.While it is important to ensure that any software application is workingas intended, it has become necessary to diligently explore the userinterface in order to adequately test common features, specialfunctionalities, logic paths, and programmatic aspects of theapplication.

SUMMARY OF THE INVENTION

The present invention relates to a method of recording the evolution ofa user interface for a software application, by monitoring imagevariations in the user interface as a user interacts with the softwareapplication executing on a computer system, detecting a user interactionwith the user interface during said monitoring, comparing a first imageof the user interface immediately prior to the user interaction with asecond image of the user interface immediately after the userinteraction to identify a region of interaction related to the userinteraction, placing a marker in a user interface evolution timeline toindicate a point in time when the user interaction occurred, and storingan image of the region of interaction and coordinates of the userinteraction in association with the marker. The coordinates of the userinteraction are preferably normalized and relative to the region ofinteraction. The comparison may be carried out by detecting that achange in the user interface has occurred using a computationallyinexpensive algorithm such as fuzzy pattern matching which compares thefirst image to the second image, and then drawing boundaries for theregion of interaction using a more computationally intensive algorithmsuch as image diffs between the first image and the second image. If theuser interaction resulted in a new user interface element which ispresent in the second image but not present in the first image, or theuser interaction resulted in removal of a user interface element whichis present in the first image but not present in the second image, theimage of the region of interaction is created using the first image. Aninteraction interval and a profile of change for the user interactioncan also be stored in association with the marker. For playbackpurposes, a GUI script can be constructed from the user interfaceevolution timeline and used to control automated execution of thesoftware application by matching a current interaction interval andcurrent profile to the interaction interval and profile of change forthe user interaction, locating a target region in an image of thecurrent user interface which corresponds to the region of interaction,and automatically applying the user interaction to the current userinterface based on the coordinates normalized and relative to the targetregion.

The above as well as additional features and advantages of the presentinvention will become apparent in the following detailed writtendescription.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be better understood, and its numerousobjects, features, and advantages made apparent to those skilled in theart by referencing the accompanying drawings.

FIG. 1 is a block diagram of a computer system programmed to carry outautomated GUI testing in accordance with one implementation of thepresent invention;

FIGS. 2A-2C are screenshots illustrating user interactions with agraphical user interface of a software application which are recorded inaccordance with one implementation of the present invention;

FIG. 3 is a pictorial representation of a user interface evolutiontimeline having markers with associated user interaction information inaccordance with one implementation of the present invention;

FIG. 4 is a chart illustrating the logical flow for creating a GUIscript in the form of a user interface evolution timeline such as thatillustrated in FIG. 3 in accordance with one implementation of thepresent invention; and

FIG. 5 is a chart illustrating the logical flow for automatic executionof a graphical interface application using a user interface evolutiontimeline such as that illustrated in FIG. 3 in accordance with oneimplementation of the present invention.

The use of the same reference symbols in different drawings indicatessimilar or identical items.

DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

Quality assurance for software applications includes rigorous testing ofthe graphical user interface (GUI). Complete manual testing of a GUI isgenerally not feasible since the number of GUI states to be verified isenormous, the number of permutations of possible inputs and events isextremely large, and there may be logic dependencies between the GUI andthe underlying application. These problems are exacerbated in regressiontesting (when a new feature is added to an application or a bug isfixed), and can lead to both false negatives and false positives.Methods have accordingly been devised for automated testing which useGUI scripts. However, creating new GUI scripts can also be verycumbersome and time-consuming.

One approach to simplifying GUI script creation is to record actual userinteractions with the application under test. Specialized softwareloaded on the same machine as the application monitors user input eventssuch as mouse button clicks, keystroke commands and text entry. Thesetools can generate messages from the application program interface (API)layer, and at the underlying graphical framework layer (e.g., X11). SuchGUI record/playback tools allow the designer or tester to capture thesteps necessary to automatically recreate GUI usage scenarios.Unfortunately, as a script attempts to automate interactions deeper inan interface, it is more likely that the automation will fail.Furthermore, for more complicated graphical applications which useentirely custom UI elements, it may be impossible to craft acomprehensive script from a user interaction record. Conventional GUItest methods lack the ability to effectively identify UI elements andinteract appropriately. While there are many offline image recognitionand comparison algorithms capable of detecting shapes and patternswithin complex images, for a real-time system such offline algorithmscannot provide the required performance. In particular, periodicallyapplying a complex image recognition algorithm to a real-time datastream may miss pertinent details within the data stream.

To solve these problems the majority of GUI record/playback automationtools limit the effectiveness of the image recognition algorithms usedto identify relevant user interface elements. The analysis is typicallyrestricted to some specific region of the image, or uses more naiveimage recognition methods. In this way a sufficiently small timeinterval can be maintained between user interface samples to catch mostrelevant events. The trade-off, however, is that less complex imagerecognition algorithms are less robust to changes in detail (e.g.,different fonts), position (e.g., different dialog placement) andperipheral state (e.g., different window decorations or desktopbackgrounds). It would, therefore, be desirable to devise an improvedmethod of GUI script creation which could more accurately mirror how auser interacts with the application irrespective of the underlyingimplementation details. It would be further advantageous if the methodcould robustly deal with differing interface element placement betweenruns of an application.

With reference now to the figures, and in particular with reference toFIG. 1, there is depicted one embodiment 10 of a computer system inwhich the present invention may be implemented to carry out automatedexecution of arbitrary graphical interface applications, particularlyfor GUI testing. Computer system 10 is a symmetric multiprocessor (SMP)system having a plurality of processors 12 a, 12 b connected to a systembus 14. System bus 14 is further connected to a combined memorycontroller/host bridge (MC/HB) 16 which provides an interface to systemmemory 18.

System memory 18 may be a local memory device or alternatively mayinclude a plurality of distributed memory devices, preferably dynamicrandom-access memory (DRAM). There may be additional structures in thememory hierarchy which are not depicted, such as on-board (L1) andsecond-level (L2) or third-level (L3) caches.

MC/HB 16 also has an interface to peripheral component interconnect(PCI) Express links 20 a, 20 b, 20 c. Each PCI Express (PCIe) link 20 a,20 b is connected to a respective PCIe adaptor 22 a, 22 b, and each PCIeadaptor 22 a, 22 b is connected to a respective input/output (I/O)device 24 a, 24 b. MC/HB 16 may additionally have an interface to an I/Obus 26 which is connected to a switch (I/O fabric) 28. Switch 28provides a fan-out for the I/O bus to a plurality of PCI links 20 d, 20e, 20 f. These PCI links are connected to more PCIe adaptors 22 c, 22 d,22 e which in turn support more I/O devices 24 c, 24 d, 24 e. The I/Odevices may include, without limitation, a keyboard, a graphicalpointing device (mouse), a microphone, a display device, speakers, apermanent storage device (hard disk drive) or an array of such storagedevices, an optical disk drive, and a network card. Each PCIe adaptorprovides an interface between the PCI link and the respective I/Odevice. MC/HB 16 provides a low latency path through which processors 12a, 12 b may access PCI devices mapped anywhere within bus memory or I/Oaddress spaces. MC/HB 16 further provides a high bandwidth path to allowthe PCI devices to access memory 18. Switch 28 may provide peer-to-peercommunications between different endpoints and this data traffic doesnot need to be forwarded to MC/HB 16 if it does not involvecache-coherent memory transfers. Switch 28 is shown as a separatelogical component but it could be integrated into MC/HB 16.

In this embodiment, PCI link 20 c connects MC/HB 16 to a serviceprocessor interface 30 to allow communications between I/O device 24 aand a service processor 32. Service processor 32 is connected toprocessors 12 a, 12 b via a JTAG interface 34, and uses an attentionline 36 which interrupts the operation of processors 12 a, 12 b. Serviceprocessor 32 may have its own local memory 38, and is connected toread-only memory (ROM) 40 which stores various program instructions forsystem startup. Service processor 32 may also have access to a hardwareoperator panel 42 to provide system status and diagnostic information.

In alternative embodiments computer system 10 may include modificationsof these hardware components or their interconnections, or additionalcomponents, so the depicted example should not be construed as implyingany architectural limitations with respect to the present invention. Theinvention may further be implemented in an equivalent cloud computingnetwork.

When computer system 10 is initially powered up, service processor 32uses JTAG interface 34 to interrogate the system (host) processors 12 a,12 b and MC/HB 16. After completing the interrogation, service processor32 acquires an inventory and topology for computer system 10. Serviceprocessor 32 then executes various tests such as built-in-self-tests(BISTs), basic assurance tests (BATs), and memory tests on thecomponents of computer system 10. Any error information for failuresdetected during the testing is reported by service processor 32 tooperator panel 42. If a valid configuration of system resources is stillpossible after taking out any components found to be faulty during thetesting then computer system 10 is allowed to proceed. Executable codeis loaded into memory 18 and service processor 32 releases hostprocessors 12 a, 12 b for execution of the program code, e.g., anoperating system (OS) which is used to launch applications and inparticular the GUI record/playback tool of the present invention whichmay be stored in a hard disk drive of the system (an I/O device 24) orother computer readable storage medium such as an optical disk (e.g., CDor DVD). While host processors 12 a, 12 b are executing program code,service processor 32 may enter a mode of monitoring and reporting anyoperating parameters or errors, such as the cooling fan speed andoperation, thermal sensors, power supply regulators, and recoverable andnon-recoverable errors reported by any of processors 12 a, 12 b, memory18, and MC/HB 16. Service processor 32 may take further action based onthe type of errors or defined thresholds.

As will be appreciated by one skilled in the art, the present inventionmay be embodied as a system, method or computer program product.Accordingly, the present invention may take the form of an entirelyhardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,the present invention may take the form of a computer program productembodied in any tangible medium of expression having computer usableprogram code embodied in the medium.

Any combination of one or more computer usable or computer readablemedia may be utilized. The computer-usable or computer-readable mediummay be, for example but not limited to, an electronic, magnetic,optical, electromagnetic, infrared, or semiconductor system, apparatus,device, or propagation medium. More specific examples (a non-exhaustivelist) of the computer-readable medium would include the following: anelectrical connection having one or more wires, a portable computerdiskette, a hard disk, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), an optical fiber, a portable compact disc read-only memory(CDROM), an optical storage device, a transmission media such as thosesupporting the Internet or an intranet, or a magnetic storage device.The computer-usable or computer-readable medium could even be paper oranother suitable medium upon which the program is printed, as theprogram can be electronically captured, via, for instance, opticalscanning of the paper or other medium, then compiled, interpreted, orotherwise processed in a suitable manner, if necessary, and then storedin a computer memory. In the context of this invention, acomputer-usable or computer-readable medium may be any medium that cancontain, store, communicate, propagate, or transport the program for useby or in connection with the instruction execution system, apparatus, ordevice. The computer-usable medium may include a propagated data signalwith the computer-usable program code embodied therewith, either inbaseband or as part of a carrier wave. The computer usable program codemay be transmitted using any appropriate medium, including but notlimited to wireless, wireline, optical fiber cable, RF, etc.

Computer program code for carrying out operations of the presentinvention may be written in any combination of one or more programminglanguages, including an object oriented programming language such asJava, Smalltalk, C++ or the like and conventional procedural programminglanguages, written for a variety of platforms such as an AIX environmentor operating systems such as Windows 7 or Linux. The program code mayexecute entirely on the user's computer, partly on the user's computer,as a stand-alone software package, partly on the user's computer andpartly on a remote computer or entirely on the remote computer orserver. In the latter scenario, the remote computer may be connected tothe user's computer through any type of network, including a local areanetwork (LAN) or a wide area network (WAN), or the connection may bemade to an external computer (for example, through the Internet using anInternet Service Provider).

The present invention is described below with reference to flowchartillustrations and/or block diagrams of methods, apparatus (systems) andcomputer program products according to embodiments of the invention. Itwill be understood that each block of the flowchart illustrations and/orblock diagrams, and combinations of blocks in the flowchartillustrations and/or block diagrams, can be implemented by computerprogram instructions. These computer program instructions may beprovided to a processor of a general purpose computer, special purposecomputer, or other programmable data processing apparatus to produce amachine, such that the instructions, which execute via the processor ofthe computer or other programmable data processing apparatus, createmeans for implementing the functions/acts specified in the flowchartand/or block diagram block or blocks.

These computer program instructions may also be stored in acomputer-readable medium that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instruction meanswhich implement the function/act specified in the flowchart and/or blockdiagram block or blocks. Such storage media excludes transitory media.

The computer program instructions may further be loaded onto a computeror other programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer implemented process such that theinstructions which execute on the computer or other programmableapparatus provide processes for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. Each block of the block diagrams and/orflowchart illustration, and combinations of blocks in the block diagramsand/or flowchart illustration, can be implemented by special purposehardware-based systems that perform the specified functions or acts, orcombinations of special purpose hardware and computer instructions.

Computer system 10 carries out program instructions for a GUI testingprocess that uses novel capture techniques to detect and record userinteractions with the software application under test. Computer system10 may also be used to automatically playback these stored userinteractions in scripted form at a later time for testing or otherpurposes. Accordingly, a program embodying the invention mayadditionally include conventional aspects of various GUI testing tools,and these details will become apparent to those skilled in the art uponreference to this disclosure.

Referring now to FIGS. 2A-2C, there is depicted an example of a userinterface for a program (software application) running on computersystem 10 which is to be tested or whose execution is to be otherwiseautomated. The user interface is displayed on a display device 24 suchas an LCD panel. In this example the software application is a documenteditor and the user interface includes a parent window 50 having agenerally rectangular frame 52 and interior features including a titlebar 54, a command bar 56, a small icon set 58, a large icon set 60, andtwo child windows 62 a, 62 b. Those skilled in the art will appreciatethat these interface features are not exclusive and the presentinvention is not limited to any specific attributes of the userinterface. The user may use various keystrokes or keystroke combinationson a keyboard to input commands which affect the user interface, and mayalso use a graphical pointing device (mouse) to control a pointingsymbol such as an arrowhead 64 for further interaction with features ofthe user interface.

Computer system 10 monitors image variations in user interface 50 as auser interacts with the program application, and detects any userinteractions. The images may be recorded using a screen capture utilityprovided by the operating system or via other means such as a remotedesktop protocol or virtual network computing. Images may alternativelybe captured using hardware devices attached to computer system 10. Theglobal, temporal variation of the image can be metrically recorded toprovide a profile of the user interface evolution in abstract terms. Awell-chosen metric is robust to changes in resolution, fonts, and evencolor variation between runs of the same application. Other minorchanges such as the location of mouse pointer 64 within the GUI can alsobe excluded. The profile can include for example a list of distinctivefeatures currently visible in the GUI such as a parent window of theapplication, any child windows, toolbars such as command bars, icon barsor status bars, pull-down or pop-up menus, dialog boxes, alarms, etc.Profiles of change can further be established by comparing thedifferences between two successive profiles (pre- and post-event).

The types of user interactions that are recorded for later playback(scripting) may optionally be limited by selectable filters, so a testermight only allow for mouse events such as mouse button clicks ordrag-and-drop while excluding keyboard events. Those skilled in the artwill appreciate however that the invention is not limited to just mouseor keyboard events but contemplates any user input device includingwithout limitation optical devices (e.g., infrared controllers), audiodevices (e.g., with voice recognition), special purpose instrumentation,or advanced touchscreen events such as sliding, spreading and squeezing.The user interactions may be detected in various manners; in anillustrative embodiment the operating system provides hooks (code whichintercepts function calls, messages or events) that can be used forevent detection.

Further to this example, the user in FIG. 2A has moved mouse pointer 64over an icon in the large icon set 60 which corresponds to a command forinserting a table in a document. FIG. 2B illustrates user interface 50immediately after the appropriate mouse button has been clicked withpointer 64 over the table icon. A dialog box 66 has now appeared foruser input regarding creation of a table within the current documentwindow (child window 62 a). Dialog box 66 may present a number ofadditional user interface features including numerical fields or radiobuttons, and command buttons such as “OK” and “Cancel”. Upon detectingthe user interaction (mouse click on the table icon), computer system 10compares the image of user interface 50 prior to the interaction (FIG.2A) with the image of user interface 50 immediately after theinteraction (FIG. 2B) to identify a region of interest related to theinteraction.

The invention preferably identifies the region of interaction in atwo-step process. First, a computationally inexpensive image recognitionalgorithm is employed to determine whether any threshold change at allhas occurred, without characterizing that change. The region ofinteraction is then pinpointed in response to the determination that theimage has changed, using a more computationally intensive imagerecognition algorithm. The simpler algorithm may be a one dimensionalimage metric to allow shorter sampling intervals. A one dimensionalimage metric is a function that transforms an image to a singlenumerical value, so the differences between two images can likewise beconverted to a single number. This fine granularity analysis can includesimple fuzzy pattern matching to identify when user interface changesoccur. When the fuzzy pattern match indicates a substantial change acomplex image recognition algorithm is triggered to draw the boundariesof the region. Image diffs can be used to identify differences betweentwo images (looping over each pixel in the first image and comparing itto the pixel in the same location in the second image). The image diffsmay be used by a two-dimensional image metric, including a 2-D fuzzymatching algorithm. In this way the invention retains the benefits ofimproved temporal granularity while also providing higher imagerecognition accuracy, yielding more robustness to user interface changesand environmental differences.

Image recognition and pattern matching algorithms may have variousparameters that can be tuned. Given a set of reference machines withdiffering environments (resolution, fonts, themes, etc.) a recordedevent sequence could be replayed on each machine and the parameter withthe minimum fuzziness could be automatically chosen to pass on to allreference systems for the initial detection of image change. Thisreduces the amount of user interaction required to produce an effectiveautomation sequence.

Given the coordinates of the user interaction, the interaction profile,and the image diffs, computer system 10 can determine the region of theuser interface that pertains to the user input event. However, theregion of change in the user interface is not necessarily the region ofinterest which is identified as being associated with that userinteraction. In the example of FIGS. 2A-2B the region of change is thedialog box 66 but that region only appeared after the user interaction(i.e., the user interaction was not with any feature of dialog box 66),so the region of interest is the parent window 52 (i.e., the userinteraction was with a feature of the parent window). Computer system 10is programmed to recognize such occurrences by examining the coordinatesof the user interaction (e.g., mouse click) relative to the region ofchange and considering whether the region of change constitutes aprofile element that was added or removed. In cases such as FIGS. 2A-2Bwhere a new profile element has just appeared and the coordinates of theuser interaction fall outside of that element, the GUI recordingfunction understands that the region of interaction will be determinedusing the image for the user interface as it existed immediately priorto the user interaction (and using the associated profile information).Similarly, if a user interface element disappears as a result of anotheruser interaction (as described further in conjunction with FIG. 2C), theregion of interaction will be determined by inspecting the imageimmediately prior to that user interaction.

Once the user input event has been detected and a region of interactionidentified, the invention places a marker in a user interface evolutiontimeline to indicate an event order or point in time when the userinteraction occurred. The type of user interaction, the coordinates ofthe user interaction, the interaction interval, and a profile of changeare then recorded in association with the marker. An image capture ofthe region of interaction (as opposed to the entire display) is alsorecorded with the marker. The coordinates can advantageously benormalized and relative to the region of interaction to account for GUIwindows which can be variably positioned and sized. Accordingly forFIGS. 2A-2B the coordinates are normalized and relative to parent window52.

This process is repeated for each interaction the user has with theinterface. Computer system 10 continues inspecting the interface imageand detecting user interactions until the parameters for the particularscript recording session are satisfied.

Continuing with the example of FIG. 2B, the user is happy with defaultsettings for table creation and simply clicks on the “OK” button ofdialog box 66. Upon clicking the “OK” button, dialog box 66 disappearsas seen in FIG. 2C. This image change is detected and computer system 10determines that the profile of change includes removal of dialog box 66.The region of interest is accordingly dialog box 66, and an image ofthat dialog box (from the previous state of the user interface) isrecorded in association with the next marker along with coordinates ofthe mouse click relative to the dialog box, the interaction interval,and the profile of change. Although the image has also changed by theaddition of a table in the letter being drafted, computer system 10recognizes that this change is not part of the user interface because itpertains to content within a data field of the already existing childwindow 62 a, as opposed to a new user interface element being addedwithin user interface 50 (there is no profile change specificallyassociated with the table's creation).

FIG. 3 illustrates a simplified schematic of the user interfaceevolution timeline 68. Markers 70 a, 70 b, 70 c have been placed atthree different times t₁, t₂, t₃, along timeline 68. According to thisimplementation of the invention each marker includes a record of thetype of user interaction, the image of the region of interaction, theinteraction coordinates, the interaction interval, and the profile ofchange. While this figure depicts only three markers, it is understoodthat the final timeline could have hundreds or even thousands of userinteractions and markers. UI evolution timeline 68 can be stored incomputer system 10 for later automation, or can be stored on a portablecomputer-readable storage medium such as a CD, DVD, or memory stick toallow transfer of the timeline for testing on a different computersystem.

Playback of user interactions with the software application beingautomated can be accomplished using a GUI script fashioned from all orany part, or selected parts, of UI evolution timeline 68. This GUIscript is then loaded onto computer system 10. The software applicationcan be launched before or after the GUI script is loaded. Computersystem 10 monitors the interface image during program operation andestablishes profiles as the interface evolves. Computer system 10compares the current timing interval and current profile to markers inUI evolution timeline 68. If the interval and profile match a marker,the associated script command is executed. The match does notnecessarily need to be perfect, e.g., the interaction interval could bewithin a predetermined range of the recorded interval, and the profilesmight have some common elements including critically defined elementsbut not all matching elements. The target region in the current image ofthe user interface is located by any convenient means, such asnormalized cross correlation against the image stored with the marker.The normalized coordinates of the interaction are applied relative tothe located region within the image, and the user interaction (mouseclick, etc.) is then activated at that point.

The present invention may be further understood with reference to thecharts of FIGS. 4 and 5 which illustrate logical flows for GUI scriptrecording and playback in accordance with one implementation of thepresent invention. In FIG. 4, the recording process 80 begins withmonitoring of the user interface image during program execution of thesoftware application 82. Computer system 10 waits for a user interaction84. If any user interaction is detected, the quick algorithm is employedto determine is the user interface has changed 86. If there userinteraction did not result in any change then computer system 10 justkeeps monitoring the usage. If the user interface has changed, theinteraction interval is computed and the region of interaction isidentified 88. The normalized, relative coordinates of the interactionare computed 90, and the profile of change is determined 92. A marker isthen placed in the UI evolution timeline at the time of the interaction94, and the interaction, region image, coordinates, interval, andprofile of changed are recorded in association with the marker 96. Theprocess repeats from the start until there is sufficient input test datato generate a GUI script 98.

In FIG. 5, the playback process 100 begins by receiving the UI evolutiontimeline or a GUI script based on such a timeline 102. Computer system10 again monitors the user interface image during automated operation ofthe software application 104. For each sample, the interval since thelast user interaction and the current user interface profile arecompared to those in the GUI script/timeline 106. When the interval andprofile match, the corresponding region of interest in the current imageis located 108 and the associated user interaction is applied at therelative coordinates 110. The playback process continues looking formatches and applying the appropriate user interactions according to theGUI script until the parameters of the automated execution are complete112.

The present invention accordingly possesses many advantages over priorart GUI recording/playback tools. The ability to effectively identify UIelements and interact appropriately allows the invention to moreaccurately mirror how a user interacts with the application irrespectiveof the underlying implementation details. The result is a robustapproach, with respect to timing, UI positioning and minor UI stylingchanges, to replaying recorded user interactions for automated executionof arbitrary graphical interface applications. The use of two differentimage processing algorithms imparts an opportunity to save computationtime and thereby allow the test framework to respond to interface eventsmore quickly.

Although the invention has been described with reference to specificembodiments, this description is not meant to be construed in a limitingsense. Various modifications of the disclosed embodiments, as well asalternative embodiments of the invention, will become apparent topersons skilled in the art upon reference to the description of theinvention. For example, while the invention has been described in thecontext of quality assurance testing, it is useful for the automatedexecution of a software application for any purpose, including but notlimited to development, demonstration, or business process automation.More generally, the invention is applicable to any system which may havehigh bandwidth multiparametric variations and correlated activities. Itis therefore contemplated that such modifications can be made withoutdeparting from the spirit or scope of the present invention as definedin the appended claims.

1. A method of recording the evolution of a user interface for asoftware application, comprising: monitoring image variations in theuser interface as a user interacts with the software applicationexecuting on a computer system; detecting a user interaction with theuser interface during said monitoring; comparing a first image of theuser interface immediately prior to the user interaction with a secondimage of the user interface immediately after the user interaction toidentify a region of interaction related to the user interaction;placing a marker in a user interface evolution timeline to indicate apoint in time when the user interaction occurred; and storing an imageof the region of interaction and coordinates of the user interaction inassociation with the marker.
 2. The method of claim 1 wherein thecoordinates of the user interaction are normalized and relative to theregion of interaction.
 3. The method of claim 1 wherein said comparingincludes: detecting that a change in the user interface has occurredusing a fuzzy pattern matching algorithm which compares the first imageto the second image; and responsive to said detecting, drawingboundaries for the region of interaction using image diffs between thefirst image and the second image.
 4. The method of claim 1, furthercomprising: determining that the user interaction resulted in a new userinterface element which is present in the second image but not presentin the first image; and responsive to said determining, creating theimage of the region of interaction using the first image.
 5. The methodof claim 1, further comprising: determining that the user interactionresulted in removal of a user interface element which is present in thefirst image but not present in the second image; and responsive to saiddetermining, creating the image of the region of interaction using thefirst image.
 6. The method of claim 1, further comprising storing aninteraction interval and a profile of change for the user interaction inassociation with the marker.
 7. The method of claim 6, furthercomprising: constructing a GUI script from the user interface evolutiontimeline; and controlling automated execution of the softwareapplication using the GUI script, wherein said controlling includesmatching a current interaction interval and current profile to theinteraction interval and profile of change for the user interaction and,responsive to said matching, locating a target region in an image of thecurrent user interface of the automatically executing softwareapplication which corresponds to the region of interaction and applyingthe user interaction to the current user interface based on thecoordinates normalized and relative to the target region. 8.-21.(canceled)
 22. A method of automating execution of a graphical interfaceapplication, comprising: receiving a metric record of prior usage of thegraphical interface application including a plurality of userinteractions with a user interface of the graphical interfaceapplication, each user interaction having a recorded region ofinteraction image, a recorded interaction interval, a recorded profileof change, and recorded interaction coordinates; monitoring imagevariations in the user interface of the graphical interface application;matching a current interaction interval and current profile for the userinterface of the graphical interface application to a recordedinteraction interval and recorded profile of change for a matching oneof the user interactions; and locating a target region in an image ofthe user interface of the graphical interface application whichcorresponds to a recorded region of interaction image for the matchinguser interaction; and automatically applying the matching userinteraction to the user interface of the graphical interface applicationusing the recorded interaction coordinates.
 23. The method of claim 22wherein the target region is located using normalized cross-correlation.24. The method of claim 22 wherein the recorded interaction coordinatesare applied normalized and relative to the target region.